Towards Endoscopic Augmented Reality for Robotically Assisted Minimally Invasive Cardiac Surgery

One of the problems tightly linked to endoscopic surgery is the fact that, because of the narrow field of view, it is sometimes quite difficult to locate the objects that can be seen through the endoscope. This is especially true in cardiac surgery, where it is difficult to not confuse two coronary arteries on a beating heart. We propose in this paper a methodology to achieve coronary localisation by augmented reality on a robotized stereoscopic endoscope. The method we propose involves five steps: making a time-variant 3D model of the beating heart using coronarography and CT-scan or MRI, calibrating the stereoscopic endoscope, reconstructing the 3D operating field, registering the operating field surface with the 3D heart model, and adding information on the endoscopic images by augmented reality. The da VinciTM surgical system was used for our first experiments with the Cardiac Surgery team at Hôpital Européen Georges Pompidou, Paris, France.